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- 77ie MAILING DATE of this communication appears on the cover sheet with the correspondence address - 

THE REPLY FILED FAILS TO PLACE THIS APPLICATION IN CONDITION FOR ALLOWANCE. 

Therefore, further action by the applicant is required to avoid the abandonment of this application. A proper reply to a final 
rejection under 37 CFR 1.113 may only be either: (1) a timely filed amendment which places the application in condition for 
allowance; (2) a timely filed Notice of Appeal (with appeal fee); or (3) a timely filed Request for Continued Examination (RCE) in 
compliance with 37 CFR 1.114. 

THE PERIOD FOR REPLY [check only a) or b)J 

a) The period for reply expires (j> months from the mailing date of the final rejection. 

b) □ In view of the early submission of the proposed reply (within two months as set forth in MPEP § 706.07 (f)), the period for reply 
expires on the mailing date of this Advisory Action, OR continues to run from the mailing date of the final rejection, whichever 
is later. In no event, however, will the statutory period for the reply expire later than SIX MONTHS from the mailing date of the final 

i rejection. 

Extensions of time may be obtained under 37 CFR 1 .136(a). The date on which the petition under 37 CFR 1 .136(a) and the appropriate 
, extension fee have been filed is the date for purposes of determining the period of extension and the corresponding amount of the fee. The 
■ appropriate extension fee under 37 CFR 1 .1 7(a) is calculated from: (1) the expiration date of the shortened statutory period for reply originally 
' set in the final Office action; or (2) as set forth in (b) above, if checked. Any reply received by the Office later than three months after the 
mailing date of the final rejection, even if timely filed, may reduce any earned patent term adjustment. See 37 CFR 1.704(b). 



fi . □ A Notice of Appeal was filed on m 



Appellant's Brief must be filed within the period set forth in 



37 CFR 1.192(a), or any extension thereof (37 CFR 1.191(d)), to avoid dismissal of the appeal. 

2. □ The proposed amendment(s) will be entered upon the timely submission of a Notice of Appeal and Appeal Brief with 

requisite fees. 

3. □ The proposed amendment(s) will not be entered because: 

(a) □ they raise new issues that would require further consideration and/or search. (See NOTE below); 

(b) □ they raise the issue of new matter. (See NOTE below); 

(c) □ they are not deemed to place the application in better form for appeal by materially reducing or simplifying the 

issues for appeal; and/or 

(d) □ they present additional claims without cancelling a corresponding number of finally rejected claims, 

NOTE: 



4. □ Applicant's reply has overcome the following rejection(s): 



5. □ Newly proposed or amended claim(s) would be allowable if submitted in 

separate, timely filed amendment cancelling the non-allowable claim(s). 

6. fiZT The a) ^affidavit, b) Dexhibit, or c) Bquest for reconsideration has been considered but does NOT place the 

application in condition for allowance because: 

7. □ The affidavit or exhibit will NOT be considered because it is not directed SOLELY to issues which were newly raised by 

the Examiner in the final rejection. 

8. For purposes of Appeal, 4he status of the claim(s) is as follows (see attached written explanation, if any): 

Claim(s) allowed: 

Claim(s) objected to: 

~Claim(s) rejected: _ 

9. □ The proposed drawing correction filed on £)l has rip has not been approved b^the Examiner. 

0. □ Note the attached Information Disclosure Statement(s) (PTO-1449) Paper No(s). 
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The two 131 Affidavits submitted by the Applicant taken together overcomes the 
rejection in the Final Rejection dated 4/9/20O1. 

Two new references have come to the Examiner attention subsequent to said Affidavit. 
They are: 

1. Grossman, Jerome H.; Plugged-in medicine; Technology Review; Cambridge; 
Vol. 97, No. 1; page 22 (+8); January 1994. 

2. Lowe, Henry J. et zl; Building a medical multimedia database system to 
integrate clinical information: an application of high-performance computing 
and communication technology; Bulletin of the Medical Library Association; 
Vol. 83; No. 1; January 1995. 
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Abstract: 

Technologies for improving and managing the production process of health care have been emerging over the 
past decade. They include clinical decision-making systems, advanced information and management-support 
systems, and multimedia communications networks. Together they may be regarded as the tools to reengineer 
medical practice: all are designed to help the health care system perform better by providing improved care at 
lower costs. As doctors and institutions pursue this vision of efficient health care based on outcomes, 
complexities arise that cry out for technological solutions. First, there is the tremendous volume of new 
information that must be processed. At the same time, providers must contend with the high degree of 
fragmentation that exists within the health care system. 

Full Text: 

Copyright Massachusetts Institute of Technology Alumni Association Jan 1 994 

When one thinks of technology in health care, the first things that usually come to mind are medical advances: gene 
therapy, laser surgery, diagnostic imaging, and the like. Indeed, this is where we have put our greatest efforts in the 
post- World War II era of biomedical science. But a different kind of health-care technology, less flashy and less likely 
to make headlines, has been emerging over the past decade, and may soon equal or surpass high-tech therapies in 
importance. These are technologies for improving and managing the "production" process of health care. They 
include clinical decision-making systems, advanced information and management-support systems, and multimedia 
communications networks. Together they may be regarded as the tools to reengineer medical 

practice: all are designed to help the health-care system perform better by providing improved care at lower costs. 

The rise of health-care management technology mirrors the shift in priorities that is gradually taking place in the 
medical community. As the public debate over health-care reform has made clear, providers have put too much 
emphasis on inputs and too little emphasis on outputs, or "outcomes." That is, we have mistakenly focused on 
delivering more and better tests and procedures instead of on improving the health, well-being, and satisfaction of 
patients. 

Under the old way of thinking, providers have not been paid to maintain or restore health; we have been paid to 
dispense units to serve. If the price of lab tests and medications goes up, patients are charged more, regardless of 
whether there is a better end product. In traditional fee-for-service payment plans, we have been rewarded for making 
the greatest use of inputs, regardless of their effect on the patient f s condition. The more we do, the more we earn. 
Accordingly, we have steadily improved the clinical tools at our disposal for diagnosing and treating disease. Yet we 
have paid much less attention to the bigger picture: how to produce the best outcomes for the patient as efficiently as 
possible. Progress toward this goal has not even been measured, let alone rewarded. 

Over the past decade, several developments have moved us toward a new way of thinking. The marketplace, spurred 
by a large employers and governmental payers, has begun demanding value by purchasing health care more 
selectively and by trying to define and measure quality. Managed-care plans like health maintenance organizations, 
which pay providers a set amount of care for a certain population for a certain period of time, have proliferated. In 
1983, Medicare introduced a payment system in which it gives hospitals a single, predetermined* sum for each 
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admission, according to the patients diagnosis-related group (DRG). As a result of these changes, providers have 
begun devoting more attention to the efficiency and effectiveness of the care they dispense. 

Under this new way of thinking, the service that medicine provides is integrated health care rather than discrete units 
of things done for patients. Depending on the patient's needs, the integrated service might take the form of routine 
health maintenance, treatment of an illness or injury, or management of a chronic disease. Providers and payers set 
price levels for their reasonable costs, and except in unique cases providers must live within these established revenue 
limits. 



Such a system dramatically changes the incentives within the delivery system. With revenues prepaid and 
predetermined, each service unit becomes an expense, not a source of income. The costs of each element of care must 
be weighed against its benefits to identify the most cost-effective process. At the same time, the outcome— the health 
and satisfaction of the patient-is measured and monitored, so that cost control works together with quality assurance 
to minimize the incentive to underutilize services. 



As doctors and institutions pursue this vision of efficient health care based on outcomes, we run into complexities that 
cry out for technological solutions. First, there is a tremendous volume of new information that must be processed. 
The already overwhelming amount of data a physician must assimilate on patients' history, condition, and treatment is 
being compounded by new layers of information on their emotional well-being and how well they function at home 
and at work. As the variables multiply, it is becoming harder for a doctor to know which ones are critical for making 
clinical decisions. Hospital administrators, meanwhile, must be able to track and organize a welter of data to 
distinguish between necessary and inappropriate services, to identify opportunities for greater efficiency, and to 
project the cost implications of substituting one form of treatment for another. 

At the same time, providers must contend with the high degree of fragmentation that exists within the health-care 
system. Communication barriers stand between the various provider units (primary care and specialist physicians, 
hospitals, laboratories, rehabilitation and nursing home facilities, and home health providers); between the providers, 
the health plans, and the payers; and between the patient and the health-care system. These barriers need to be 
m in imized so that caregivers can consult together effectively, and so that information about a patient can reach 
everyone who should know a s quickly as possible. Different departments within a hospital also need to be able to 
pool information to help document how well the institution is serving its patients and payers. 

To relieve these problems, health-care providers are turning to sophisticated information and networking technologies. 
Computerized information systems can help clinicians make better decisions about how to treat individuals and 
groups of patients, as well as help administrators make decisions about the resources and processes that affect that 
treatment. Networks and other communications technologies can help fuse the many components and players into a 
seamless fabric of care. Such systems are gradually transforming the way we practice, the way we organize and 
manage care, the way providers relate to each other, and the way patients relate to the medical community. 

CONSULTING COMPUTERS 



The work of physicians involves endless testing of hypotheses and posing of questions: What is the patient's 
diagnosis? What test should I do to confirm the diagnosis! What is the best treatment? How is the treatment working? 

In arriving at a decision and recommendation, physicians of course rely on their own judgment and experience, on the 
relevant literature, and on established clinical guidelines for treating different types of patients. But valuable as these 
resources are, there are still occasions when the answers are not clear. In an emergency, for example, the doctor may 
not have time to refer to the literature or search for the latest guideline. And the clinical guidelines for a certain 
condition, even if readily available, might not take into account the peculiarities of the case at hand: the patient's age, 
sex, and race; the stage of disease; the presence of other medical conditions; and special risk factors-like allergies. An 
error can prove costly, both in dollars and, more important, in the patient's health and survival. 

Physicians at my hospital, New England Medical Center, are working on two types of compute-assisted "decision 
support" techniques for just such circumstances. One approach is intended to help physicians make accurate diagnoses 
in emergencies. The other will help determine the best course of treatment for individuals or groups. 

In addressing emergency decision making, our goal is to develop accurate and easy-to-use instruments that can isolate 
the most important variables and calculate the probability that the patient is or is not going through a crisis that calls 
for some specified action. One such device developed by our researchers and now in trial use is the Acute Cardiac 
Ischemia Time-Insensitive Predictive Instrument. ACI-TIPI (pronounced "A.C.I.-tippy") helps emergency-room 
physicians quickly determine the probability that a suspected heart attack is real. A correct diagnosis is of paramount 
importance, both medically and economically. Each year, 1 .5 million patients are admitted to coronary-care units, yet 
fewer than half are actually experiencing, or about to experience, a heart attack. These unnecessary admissions cost 
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over $3 billion a year. More costly in lives is the opposite sort of mistake: some 20,000 heart attack victims each year 
are sent home from emergency rooms without treatment. 

In the emergency room, an electrocardiograph is outfitted with a special ACI-TIPI computer program. The physician 
enters a few items of information, such as the. patient's age and gender and whether chest pain is the chief complaint, 
and generates an electrocardiogram (EKG). The program analyzes these factors along with three EKG-related 
features, and then computes the probability—from 0 to 100 percent— that the patient is suffering from acute cardiac 
ischemia (a severe reduction of blood flow to the heart) and prints it on the EKG. The entire procedure can be 
completed within minutes of the patient's arrival, instead of the several hours it takes to perform a blood test that can 
confirm a heart attack. That time difference can be critical. A two-year study of an ACI-TIPI prototype found that its 
use reduced unnecessary coronary-care admissions by 30 percent. Projected to ever; U.S. hospital, that figure would 
translate into 250,000 fewer admissions each year, as well as a saving of perhaps $1 billion in unnecessary health-care 
costs. The latest version of ACI-TIPI, which is easier to use than the prototype, is now undergoing clinical trials on 
some 15,000 patients at 10 hospitals around the country. 

For non-emergency care, researchers at New England Medical Center are collaborating with MITs Laboratory of 
Computer Science to develop computerized "expert systems" that help doctors make the most sensible treatment 
decisions in complex cases. Some of these tools quantify the risks and benefits-both clinical and economic—of 
different strategies for treating a given condition. They are proving useful for resolving dilemmas in treating coronary 
artery disease and diabetes, performing heart transplants, diagnosing prenatal and neonatal conditions, and other 
situations. These systems offer physicians choices in the form of a "decision tree," allowing them to see the possible 
ramifications of each step: If a patient receives surgery instead of medication, what is the probability that 
complications will set in? If the complications are treated with drugs, what is the probability that the patient will live? 
If the patient lives, what is the probability that he or she will suffer from a chronic impairment? And so on. 

Such decision trees can be tailored to include a patient's individual preferences and risk factors such as age, sex, and 
the presence of secondary conditions. For example, a decision tree we developed in collaboration with Duke 
University examines the choice among bypass surgery, less intensive angioplasty, and medical therapy for chronic 
stable angina. Such models might take into account that the patient would rather live with some pain than receive a 
treatment that carried a risk of stroke. Other models being developed jointly at New England Medical Center and MIT 
are designed to help physicians predict a patient's response to drugs. 

THE PLUGGED-IN PHYSICIAN 

Beyond helping doctors make decisions in unique or difficult situations, information technology can serve as an 
invaluable aid to physicians in routine practice. In particular, the automated collection, analysis, and transmission of 
data allows doctors to get the information they need when they need it. The result is better clinical care, more efficient 
and patient-friendly service, and higher job satisfaction for caregivers. 

The physician stands at the center of a swirl of information— from patients, from labs and radiology departments, from 
specialists. Today, most transactions in health care do not exist in any computerized system, and the systems that do 
contain such information are not integrated. But in the emerging computerized health-care environment, information 
from one office or lab can be flagged and delivered to another office or lab according to rules and criteria that the 
users themselves have written. 

In the traditional, nonintegrated environment, test results remain in a lab's computer until a batched group is 
transmitted to another computer, or else individual results are manually flagged and reported by phone or fax, 
whereupon the message waits until the physician can be found. The latest "intelligent message routing" systems offer 
physicians a menu of rules and options: v/bich results should be routinely transmitted, what kinds of readings warrant 
an immediate notification, where else results should be sent. If a test result is required before a patient can be 
discharged, for example, a doctor might order that this information be phoned in immediately and that a specialist 
receive a copy. The intelligent message routing system screens lab transactions, flags the ones meeting the defined 
rules, and transmits information to the requested parties, whether by electronic mail, fax, or paging device. The rules 
can also require that receipt of the information be acknowledged, or that the message be repeated at defined intervals. 

For intelligent message routing to work on a large scale, the health-care system must become far more computerized. 
Just as important, systems in different locations must be able to communicate. Fortunately, we no longer need to rely 
on expensive proprietary solutions in which systems can be integrated only if the hospital works with a single vendor. 
The trend toward open, nonproprietary standards— such as Simple Mail Transport Protocol and Dynamic Data 
Exchange-allows us to create software that integrates bits and pieces of the health-care operation as needed. 

MANAGING THE PROCESS OF CARE 



3of6 



mm/09/1999 5:25 PM 



Document http://proquestumi.cx)m/pqdweb?...Did=000000000829011&Mtd=2&Fmt=3 

The new combination of rules-based technology and standard interfaces between systems can help hospitals monitor 
and improve the overall efficiency and effectiveness of the care they deliver. In the past, each component of a hospital 
* information system-general ledger, payroll, billing, and medical records— would collect and report its own data. The 
new systems, in contrast, link these types of data together to show how each unit of service affects the overall cost and 
quality of care. 

Since the 1980s, New England Medical Center has been using systems to analyze the course of treatment for different 
groups of patients. For example, we can lump together all the admissions for diabetes and track these patients' care: 
length of stay, tests and procedures performed, costs, and clinical outcomes like readmissions. We can look at subsets 
of patients by age, other medical conditions, or physician. In the process, we might find that extra lab tests add no 
value, that one physician's patients tend to stay longer without any difference in outcomes, or that patients who 
received one medication did just as well as others who were given a more costly drug. This information is valuable for 
clinicians and managers alike. 

More recently, we have applied rules-based processing to define ideal clinical protocols that providers should be 
following. The protocols may govern the overall process of care-for example, what should be done to a certain type 
of patient on day one of hospitalization, day two, day three, on through discharge. Or they can govern specific 
components of care-recommending, say, that antibiotics not be used for more than two days after surgery unless an 
infection is diagnosed. We then write rules telling the system to flag and report variations from these protocols, giving 
staff the opportunity to intervene. We can also monitor retrospectively how actual performance compared with the 
defined protocol, or how outcomes changed as a result of managing care according to the protocol. Armed with this 
information, we can alter our practices as appropriate. 

One information system known as HELP (for "health evaluation through logical processing"), developed at Salt Lake 
City's Latter Day Saints Hospital, has been used to improve the way the hospital administers antibiotics. The system 
triggers reminders to give antibiotics to certain patients before surgery. Similarly, it issues automatic stop orders for 
patients who appear to have been left on antibiotic therapy too long, and notifies the pharmacist and the physician. 
When an infection is suspected but the agent is not known, the computer helps the physician choose the most effective 
antibiotic; using its own statistical analysis, HELP has selected the right antibiotic 94 percent of the time, a better rate 
than that achieved by physicians. The system also identifies cases where a less expensive antibiotic can be substituted 
for the one prescribed. 

In the future, we hope to incorporate a wider variety of data into these systems. A notable example is information on 
patients' emotional well-being and ability to function as reported in questionnaires at various points during and after 
the care process. This is a new and valuable measure of how well we are serving our patients. We also need to include 
data from settings other than the hospital— such as doctors' offices, labs, and outpatient clinics— in order to model and 
analyze the process of care across a full episode of illness. Over time we hope to integrate information from other 
institutions and from the research literature, so that users have more material from which to draw. 



TYING IT ALL TOGETHER 



Just as integrating different sources of information can help institutions refine their practices, it can also reduce the 
fragmentation among the many players involved in a patient's care and treatment. The challenge is to bridge the 
barriers of space and time to meet the needs of individual patients and to make smooth and efficient processes routine. 
Our goal is a seamless health-care system whose components work in harmony to respond to the needs of patients, 
regardless of where they happen to be within the system, and even if they are at home. 

To a degree, the necessary bridges can be built with existing information technology. Intelligent message routing, for 
example, can be extended to include not just different units of a hospital but other organizations such as the payer or a 
nursing home. Referral forms from an internist can be delivered to the specialist and the managed care plan at the 
same time. And hospital reports on the eve of discharge can be electronically mailed to the recuperative facility or the 
home health agency that will follow the patient. 

Transmitting information across sites is more difficult than within a single institution, but the necessary infrastructure 
is being built. One breakthrough that will affect all industrial fields, including health care, is the introduction of 
asynchronous transfer mode, or ATM, a switching technology that breaks digital transmissions down into small, 
standardized "cells," several of which can be processed at once. With ATM, vast amounts of data will be transmitted 
at higher speeds and lower costs than is now possible. Such advances not only will allow information to flow easily 
across distances but also will accommodate multimedia communications— the transmission of voice, data, still images, 
and video. 

Using multimedia communications networks, doctors will be able to view clinical information and consult together 
anywhere, anytime. A physician in one location could participate in a patient exam, or even an operation, done by 
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another physician someplace else. Such networks will also connect facilities, making equipment at a teaching hospital 
available to a community hospital. A number of demonstration "telemedicine" systems, sponsored by 
telecommunications companies with the participation of health-care institutions and providers, are already in the 
works. Bell South's planned North Carolina Information Network, for example, will put rural clinics in touch with 
specialists at research hospitals, allowing them to communicate by voice and video and share computer data and 
graphics. With this sort of technology, we will be able to eliminate much of the physical movement of caregivers and 
patients that now takes place and at the same time improve the quality and cost-effectiveness of care. 

The next step is to link the patient at home to the health-care system. Since the 1980s, institutions have attempted to 
contain costs by cutting down on inpatient care. But for patients with chronic illnesses requiring frequent medical 
attention, and no less for their families, managing at home can be a daunting task. 

In 1991, New England Medical Center and IBM Research launched a joint effort to develop an interactive 
home-based computer support system for the care of children with leukemia. Prototypes are now being installed. The 
system is meant to serve as a user-friendly educational resource and to provide clinicians with progress reports 
directly from the patient. The unit incorporates videos showing how to change bandages or care for catheters, as well 
as videos of different families and caregivers discussing clinical and emotional aspects of the illness. The computer is 
connected to the hospital through electronic mail. The unit is easy to use, with touch screens and directories for 
gaining access to stored information on symptoms, technical aspects of care, or emotional and family issues. Planned 
for the future are an electronic patient-support network that will link patient's two have similar conditions, a 
hospital-based bulletin system for news of important developments, and full-motion video transmissions between 
doctor and patient. 

This type of home interface provides numerous benefits. It gives families the information they need to participate in 
the care process more directly . It reduces the disruptions to family life by lowering the number of trips to the hospital 
or physician's office. It establishes a feedback loop between home and caregivers for monitoring the patient's physical 
and emotional state and compliance with the prescribed treatment, letting staff use their time more efficiently. And 
contrary to what one might expect, the technology actually humanizes care by addressing the emotional needs of the 
patient and family, and by easing the family's anxieties about tending to their ill member. 

The link between the homecare patient and the caregiver will soon become even stronger: schemes are now under 
development for providing patients with instruments connected to a home computer that is in turn linked to an 
information network. If the patient needs regular electrocardiograms, a mobile machine at home would transmit the 
readings in real time to the EKG lab and the physician's office. A patient whose blood pressure must be monitored 
could insert one arm into a cuff attached to a computer, which would immediately read results into the medical record 
and flag any exceptional readings for the physician's urgent attention. Technologies like these will be a major step 
toward establishing the continuity of care that has eluded the medical profession for so long. 

OVERCOMING OBSTACLES 

Like virtually any effort to change institutions and ways of operating, the transformation of health care through 
information technology is encountering some obstacles. One is cost. For many hospitals and physician practices, 
sophisticated information and networking systems are simply too expensive. Fortunately, prices are coming down as 
the underlying technologies mature. Just as we can purchase high-powered personal computers today at prices that 
were unimaginable a decade ago, information technology will continue to undergo an exponential increase in 
performance with an exponential decrease in costs. 

More good news is that organizations able to afford these technologies will find their investment quickly repaid. In 
one recent study (conducted in Minnesota and Virginia by the Tiber Group, a Chicago-based consulting firm), 
electronic filing of health-care claims and other paperwork was found to yield annual savings ranging from $30799 
for small hospitals to $1.4 million for larger hospitals, and from $13,000 for small physician practices to $183,000 for 
larger practices. Start-up costs were only $21,000 to $12,000 for hospitals and $4,000 to $13,000 for medical 
practices. The authors concluded that conducting 85 percent of all transactions electronically would save the 
health-care industry $4.7 billion a year. As more studies show the financial benefits of new information systems, the 
expense will be easier to justify. 

But technological change also runs up against human barriers. At first people may be intimidated by new computer 
systems, regardless of how user-friendly they may be. In some instances the new information systems will change the 
essence of people's jobs, for better or worse. For example, hospital staff who monitor the quality and cost of care now 
spend much of their time poring through reports to identify incidents that fail to meet specified criteria. With new 
systems that automatically flag variances, the staff occupy themselves with the more challenging task of analyzing the 
causes, effects, and solutions. 
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Designing and installing new systems must be a cooperative venture between developer and user. As much as 
possible, the new products should take into account the user's needs and habits. The ACI-TIPI system for identifying 
heart attacks, for example, was originally designed in the form of a calculator, which is not a standard part of the 
physician's "toolkit." Incorporating the system into an electrocardiograph made it more palatable. 

In the past, the conservatism of the health-care community has caused it to lag behind other industries in applying 
information technology to help manage production processes. But now we must surge ahead. The Congressional 
Research Service has estimated that about half the growth in health-care expenditures is controllable-that is, 
unrelated to general inflation or population changes. As the emphasis in medicine shifts from inputs to outcomes, the 
incentives are growing for physicians to provide high-quality, cost-effective care. Advances in information and 
networking technology can accelerate this trend by helping care providers determine the most direct, most effective, 
and least costly means of diagnosing and treating patients. Such technologies offer the tools to redefine and restructure 
health care in our time. 



Reproduced with permission of the copyright owner. Further reproduction or distribution is prohibited without permission. 
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The rapid growth of diagnostic-imaging technologies over the past 
two decades has dramatically increased the amount of nontextual data 
generated in clinical medicine. The architecture of traditional, text- 
oriented, clinical information systems has made the integration of 
digitized clinical images with the patient record problematic. Systems 
for the classification, retrieval, and integration of clinical images are 
in their infancy. Recent advances in high-performance computing, 
imaging, and networking technology now make it technologically 
and economically feasible to develop an integrated, multimedia, 
electronic patient record. As part of The National Library of 
Medicine's Biomedical Applications of High-Performance Computing 
and Communications program, we plan to develop Image Engine, a 
prototype microcomputer-based system for the storage, retrieval, 
integration, and sharing of a wide range of clinically important 
digital images. Images stored in the Image Engine database will be 
indexed and organized using the Unified Medical Language System 
Metathesaurus and will be dynamically linked to data in a text-based, 
clinical information system. We will evaluate Image Engine by 
initially implementing it in three clinical domains (oncology, 
gastroenterology, and clinical pathology) at the University of 
Pittsburgh Medical Center. 



BACKGROUND 

Patient care generates a large amount of text-based 
data. The sheer volume of this clinical data and its 
increasing importance to health care activities has 
begun a trend toward the use of computerized clinical 
information systems [1], Many of these systems are 
entirely or largely text oriented. However, text is only 
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one medium through which clinical information is 
recorded and communicated. 

The rapid growth of diagnostic imaging technol- 
ogies over the past two decades has dramatically in- 
creased the amount of nontextual data generated in 
clinical medicine. Imaging technologies are essential 
to the modern practice of clinical medicine [2]. Tra- 
ditional radiological and nuclear medicine images are 
now complemented by computerized tomography 
(CT), magnetic resonance imaging (MTU), positron 
emission tomography (PET), ultrasonography, and 
endoscopy-generated image data. Health care pro- 
viders use such images routinely to make clinical de- 
cisions. Furthermore, many medical specialties such 
as pathology, dermatology, and ophthalmology gen- 
erate a large number of clinically important images. 

Though reports on these images are added to the 
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patient record, the images themselves are usually dif- 
ficult for the clinician to access and often impossible 
to integrate with other relevant clinical data [3], Sys- 
tems for the classification, retrieval, and integration 
of clinical images are Ln their infancy [4]. Traditional 
picture archival and communications systems (PACS)' 
[5-7] are generally expensive, monolithic solutions 
thac 6erve primarily the needs of radiologists and are 
often not well integrated with the patient record. The 
new clinical imaging technologies demand innova- 
tive medical image database model6 that can integrate 
all patient data. Such systems may improve the qual- 
ity of patient care [8-9], increase the patient's involve- 
ment in clinical decision making [10-11], and produce 
significant new medical knowledge [12]. 

The architecture of traditional, text-oriented, clin- 
ical information systems has made the integration of 
digitized clinical images with the patient record prob- 
lematic. This lack of integration leads to a fragmen- 
tation of the patient record, which hinders the phy- 
sician's ability to synthesize all the data relevant to 
clinical decision making. In part, this partitioning of 
textual and nontextual patient data reflects the tech- 
nological heritage of the traditional patient record. 
Until recently, the technology required to integrate 
textual and image-based clinical information was pro- 
hibitively expensive or nonexistent. Recent advances 
in high-performance computer, imaging, and net- 
working technology now make it technologically and 
economically feasible to create a truly integrated, 
multimedia, electronic patient record linking digi- 
tized clinical images from a wide variety of sources 
with the traditional, text-based, medical record. 

BEYOND PACS 

Although PACS [13-14] have been an active area of 
research and development for almost twenty years 
[15-16], much of that work has been within the do- 
main of radiology and has focused on expensive in- 
stitutional systems [17]. Some involved with PACS 
research feel that if the fundamental idea is to pros- 
per, it must expand its domain from radiology and 
radiologist to a broad range of clinically important 
images and the majority of physicians practicing both 
inside and outside the hospital environment [18], In 
addition, given the current fiscal outlook for health 
care delivery, we must seek ways to reduce the im- 
plementation costs of integrated clinical image deliv- 
ery systems. 

The integration of clinical images with the textual 
clinical record has been an area of increasing interest 
over the past few years. Although a number of in- 
tegrated PACS systems axe described in the literature 
[19-20], one of the most innovative implementations 
of integration has been in the Department of Veterans 
Affairs (VA) Decentralized Hospital Computer Pro- 
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gram (DHCP) [21-23]. This system, developed at the 
Washington Information Systems Center of the VA, 
is a "distributed imaging system that provides image 
management and communications functionality as an 
integral part of its existing integrated hospital infor- 
mation system" [24]. The DHCP is an object-oriented 
system based around the MUMPS VA File Manager 
database using UN1X/X Windows or Microsoft Win- 
dows workstations and multiple image servers. It goes 
beyond the traditional PACS radiology model by 
"handling a variety of medical images including car- 
diology studies, microscopic pathology slides and en- 
doscopic examinations" [25]. 

There is currently a confluence of developments 
that makes it technologically and financially feasible 
to implement institutional systems integrating digi- 
tized clinical images from a wide variety of sources 
with the traditional medical record. These innova- 
tions include inexpensive microcomputer worksta- 
tions with the processing power, memory, and dis- 
play characteristics necessary to permit real-time 
decompression and display of high-resolution digital 
still images and digital video. Image compression 
technology is now widely and inexpensively avail- 
able, and international compression standards are 
emerging [26]. Digital video technology is now an 
integrated part of many microcomputer operating 
systems and permits us for the firet time to display 
real-time, on-screen, color-video sequences using 
widely available and affordable computers. Network- 
ing advances increasingly supply the necessary band- 
width for institutional transport of compressed dig- 
ital images. 

Image compression will be an important enabling 
technology in the development of integrated, mul- 
timedia clinical information systems. Even with cur- 
rent advances in affordable computer hardware, the 
large size of uncompressed images would preclude 
workable, large-scale systems. For example, one min- 
ute of digital video (480 by 640 pixels, 24-bit color at 
30 frames per second) requires 1.5 gigabytes of stor- 
age, while a single 480 by 640 pixel, 24-bit color still 
image is approximately one megabyte in size. Clinical 
information systems working with uncompressed data 
of this scale would rapidly fill even today's gigabyte 
storage devices and jam most institutional networks. 

IMAGE ENGINE 

Our Biomedical Applications of High-Performance 
Computing and Communications (HPCC) contract 
from the National Library of Medicine (NLM) will 
allow us to further develop Image Engine, a prototype 
microcomputer-based system for the storage, retriev- 
al, integration, and sharing of a wide range of clin- 
ically important digital images [27], Images stored in 
the Image Engine database will be indexed and or- 
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ganized using the Unified Medical Language System 
Metathesaurus and will be dynamically linked to data 
in a text-based, clinical information system. We will 
evaluate Image Engine by initially implementing it 
in three clinical domains at the University of Pitts- 
burgh Medical Center (UPMC): oncology, gastroen- 
terology, and clinical pathology. 

The Image Engine System architecture consists of 
three layers: the Server Layer/ the Object Database 
Layer and the Image Browser Layer. In addition, Im- 
age Engine will make use of two network-based ser- 
vices: the Medical ARchival System (MARS) clinical 
information system and the Probabilistic Indexing 
(Pindex) server. 

THE MARS CLINICAL 
INFORMATION SYSTEM 

MARS i6 a text-based / clinical information system de- 
veloped by Dr. John Vries and Russell Yount at UPMC 
[28]. MARS has been in operation at UPMC for five 
years. It contains approximately 3.2 million whole- 
text, word-indexed, clinical records. These document 
records contain the full text of patient histories and 
physicals; operative and procedure notes from mul- 
tiple clinical specialties; discharge summaries; labo- 
ratory results; and reports from the pharmacy, mi- 
crobiology, pathology, and radiology departments. 
The data stored on MARS includes 80% of all infor- 
mation generated at UPMC and all information pro- 
duced by the Central Transcription Service for 
UPMC's hospitals and outpatient clinics. More than 
2,500 registered users retrieve an average of 5,000 
reports each day, and these numbers are growing 
steadily. MARS is available twenty-four hours per 
day. It is anticipated that by 1995, MARS will capture 
most of the clinical information generated at UPMC, 

THE PINDEX INDEXING SYSTEM 

We have developed a system called Pindex that takes 
as input a string of free text and returns an associated 
list of Medical Subject Heading (MeSH) terms that 
are each annotated with a probability of relevance. 
The development of Pindex has been supported in 
large part by the NLM UMLS project. We are modi- 
fying and extending Pindex for the task of indexing 
images. We plan to use this modified version of Pin- 
dex to index medical images based on their free-text 
descriptions and to assist users retrieving images giv- 
en free-text input from the user about the type of 
images that are desired. 

Pindex works as follows. A simple parser, P, con- 
verts free text input into a set of word phrases, which 
we denote as S< Pindex has a large table, T, that as- 
sociates word phrases with MeSH terms. Each asso- 
ciation between a phrase and a term has an attached 



probability. Let U denote the MeSH terms that are 
associated with one or more phrases in S. We attach 
to each term in U the maximum probability it has in 
association with any phrase in S (as given by table 
T). 

If the free text is a description of an image, then 
the terms in U can be used to index the image. If the 
free text is a user query, then we can sort the terms 
in U in descending order of their attached probabil- 
ities. The user can choose terms from this sorted list 
to construct a search expression for images of interest. 

Table T is constructed as follows. A large group of 
MEDLINE articles is used as the training set. For each 
MEDLINE article, A { , we apply parser P to the text in 
the title and abstract of the article to derive a set of 
word phrases, S,. Let ii, denote the main MeSH head- 
ings assigned to article A k in MEDLINE. In A if we 
view each phrase in S, as co-occurring once with each 
MeSH term in L7 ( , We update table T to reflect the 
phrase-term co-occurrences for article A { and phrase 
occurrences in article A h After all articles are pro- 
cessed, table T contains the number of occurrences of 
each phrase that was encountered in the MEDLINE 
training articles and the number of co-occurrences of 
each phrase-term pair encountered. From these sta- 
tistics, we compute the probability of a MeSH term, 
u, given a phrase, $, as the total number of co-occur- 
rences of s and u divided by the total number of 
occurrences of s. To avoid misleading probability es- 
timates due to small sample sizes, we require that the 
value of s be greater than a ininimum threshold. 

The terms in table T need not be limited to MeSH 
terms. In general, we can construct the table using 
any training data that contain free-text descriptions 
that are annotated with controlled vocabulary terms. 
Thus, for example, in addition to using MEDLINE 
training data, one could use training data based on 
surgical pathology reports in the MARS system for 
which online SNOMED codes are available. Our over- 
all focus will be on using training data that contain 
terms within the UMLS Metathesaurus. 

THE IMAGE ENGINE SERVER LAYER 

The Image Engine Server Layer will consist of a ded- 
icated server computer and a number of gigabyte range 
hard disks connected to UPMC's high-speed data net- 
work. These disks will store digitized, compressed 
clinical images. Still digital images will be stored in 
the PICT format and digital video images as Quick- 
time files. PICT files will be compressed using the 
International Standards Organization's (ISO) Joint 
Photographic Experts Group (JPEG) still image com- 
pression algorithm [29-30], Digital video files will be 
compressed with the proposed ISO Motion Picture 
Experts Group (MPEG) video compression scheme [31]. 
Storing images in these widely supported and well- 
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documented formats will facilitate cross-platform de- 
velopment and image sharing. We plan to support 
image translation between a number of standard im^ 
age interchange formats. 

One of the issues we will address with image do- 
main experts ia defining the optimal compression 
characteristics for different clinical image types. Im- 
ages will be automatically compressed as they are 
added to the server database and decompressed on 
arrival at a workstation running the Image Engine 
Browser software. Image compression reduces disk 
storage requirements and reduces transfer time across 
the network. While the ISO standard JPEG is lossy at 
compression ratios above 2:1, it is often capable of 
compressing a wide range of still image types at com- 
pression ratios of 10:1 to 24:1 without detectable loss 
of image quality. (Lossless image compression means 
that although the storage requirements of an image 
are reduced and the image is therefore compressed, 
the data set for that image remains unchanged. Lossy 
image compression, on the other hand, means that in 
the process of compressing an image, some of the 
original image data set is irreversibly lost.) For ex- 
ample, formal clinical evaluation of JPEG compressed 
chest x-rays found a compression ratio of 20:1 or less 
to be acceptable [32]. The optimal degree of com- 
pression applied to any image type is highly depen- 
dent upon the nature of the image [33] and will have 
to be determined over time in consultation with im- 
age domain experts. 

THE IMAGE ENGINE OBJECT 
DATABASE LAYER 

The Object Database Layer uses an object-oriented 
database model to represent the images 6tored on the 
Image Engine Server [34]. This may be viewed as a 
virtual database, in that it integrates data stored both 
locally and (by reference) on other systems [35]. This 
approach has a number of advantages, including 
avoiding storage replication and problems with data 
version inconsistencies. This model also allows for 
future integration with other clinical information 
systems. 

The Image Engine Object Database will contain ob- 
ject-oriented representations of the images stored on 
the server. This database structure will support mul- 
tiple independent index files. Each image object rec- 
ord will have a unique identifier. Object property 
values will be indexed as either text (inverted word- 
stem index based upon the Porter Algorithm [36]) or 
data (indexed on full property value). In addition, we 
are interested in experimenting with data by refer- 
ence models (virtual data existing in other database 
systems). Multiple indexes based upon property val- 
ues will allow for rapid database searching on many 
independent criteria. 
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Figure 1 outlines the major elements of the Image 
Engine Database and its relationship to MARS and 
Pindex. The image itself is stored in the server layer 
and has an object representation in the Image Engine 
Object Database. Each image object has a set of prop- 
erty values. For simplicity, only some of the image 
objects property slots (e.g., Image Reference, Patient 
ID) are shown. All image objects in the database will 
be indexed by multiple property keys in the Image 
Engine Index. These property keys point to both the 
Image Object and the image itself. Thus the index can 
be used to find a specific image or a set of images 
meeting some specification. Property values can also 
point to data stored on MARS (i.e., data by reference). 
For example, the Patient ID property points implicitly 
to all the MARS documents for that patient. Combi- 
nations of properties can define subsets of MARS data; 
for example, all pathology reports for a given patient 
ID between date X and date Y. Pindex is shown in 
this figure in its role as an automatic indexing tool, 
taking document input from MARS and sending 
Metathesaujus terms to the Object Database. How- 
ever, Pindex could also aid data retrieval by taking 
text from a user query and returning suggested Meta- 
thesaurus terms for use in image retrieval. 

THE IMAGE ENGINE BROWSER LAYER 

The Image Engine Browser (Figure 2) is a client ap- 
plication that allows one to interact with the Object 
Database through an easy-to-use graphical user in- 
terface. The Browser communicates with the Object 
Database over the network and uses a client-server 
model to request and retrieve image and patient data. 
Currently the project's workstations are based on Ap- 
ple Power Macintosh 7100/66 RISC systems with 
twenty-four megabytes of RAM and twenty-inch, 
1,152 by 870 pixel resolution color monitors using 
accelerated, twenty-four-bit color video cards. 

The user may search for image subsets using com- 
binations of image properties, including Metathesau- 
rus terms. Retrieved subset summaries of images can 
be viewed as either a scrollable list of thumbnail im- 
ages (100 by 100 pixel scaled, 24-bit color images) or 
a text list of object identifiers (image name, type, pa- 
tient ID, and so forth). Images can be selected and 
viewed at full or scaled size on the computer display 
in resizable, scrollable windows. Multiple images can 
be viewed simultaneously. Image information text 
can be viewed simultaneously with images. Retrieved 
image sets can be sorted and displayed on a number 
of criteria. 

Digital video images can be displayed on screen 
and controlled with videotape-like features (e.g., re- 
verse, fast-forward, still frame, and reverse /forward 
frame capabilities). Users can convert digital video 
frames to digital still images. Given the large size of 
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Figure 1 

Structure and relationships of Ima^e Engine Object Database 
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even compressed digital video sets, we wish to de- 
termine whether endoscopy data, for example, stored 
in short video segments have any clinical advantage 
over selected digitized still images extracted from the 
video record. 

The browser will provide a set of basic image-pro- 
cessing functions to support image enhancement/ im- 
age formal translation, and feature measurements. In 
addition, the Image Engine Browser will communi- 
cate with and pass images to external applications 
such as image processors and electronic mail clients. 
Image and patient data from MARS will also be viewed 
within the Browser. The Image Engine architecture 
will facilitate the future integration of clinical deci- 



sion resources such as MEDLINE, FDQ, and clinical 
guidelines. 

NETWORKING PROTOCOLS 

The Image Engine server will be connected to the 
University Medical Center's network via a high-speed 
Ethernet connector and will initially communicate 
with Image Engine Client workstations at UPMC via 
a proprietary message-passing scheme using Apple 
Computer's Ethertalk networking protocol. We plan 
to switch to the standard transmission-control pro- 
tocol/Internet protocol (TCP/IP) at a later phase of 
this project. 
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Figure 2 

Image Engine Thumbnail Browser 
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The UPMC data network consists of a basic back- 
bone of three Fiber Digital Device Interface (FDDI) 
rings using fiber-optic cable with fiber-optic limbs to 
collapsed Ethernet backbones in each of the constit- 
uent hospitals and clinics of the medical center. Fiber- 
optic cable now extends to most hospital and clinic 
floors with standard lOBaseT cabling to individual 
workstations. The current average data transfer rate 
is ten megabits per second/ increasing to a maximum 
of 100 megabits per second in the near future, with 
the installation of Level-5 cabling. The current sup- 
ported routing protocols include TCP/IP, Novell's 
Internetwork Packet Exchange (IPX), and AppleTalk. 
Current bridging protocols include Digital Equip- 
ment's Local Area Transport (LAT) and Local Area 
Storage Transport (LAST), and Maintenance Opera- 
tions Protocol (MOP). 

DYNAMIC LINKS TO A TEXT-BASED 
CLINICAL INFORMATION SYSTEM 

The MARS database system supports a scripting lan- 
guage called ESP, which we will use to implement 



dynamic links between clinical images in the Image 
Engine database and associated patient information 
in the MARS database. Using ESP, an application on 
the UPMC data network, can establish a TCP/IP con- 
nection to the MARS ESP server and transmit MARS 
queries written in the ESP language. MARS patient 
data retrieved by an ESP query is returned over the 
network to the requesting application in a format 
defined in the ESP query script. ESP, therefore, allows 
an application to become a MARS client. 

The MARS ESP query language is still under de- 
velopment, and we anticipate that our work on dy- 
namically linking Image Engine objects to MARS may 
suggest, new features and enhancements to the ESP 
language and server. In particular, we are interested 
in exploring the use of "shortcut" ESP queries that 
specify exactly which documents should be retrieved. 
This will give us the capability of storing sparse ESP 
queries in Image Object property slots as data by ref- 
erence. If this approach works, we could expand the 
Image Object Database to contain image objects that 
know how to retrieve information about themselves 
from other clinical information systems without any 
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user interactions. Such objects could "stack" potential 
data items from MARS, based upon their past expe^ 
rience with particular users or past sequences of ao 
tions leading to data retrieval. For example, such an 
object could anticipate that if a user views a colonos- 
copy report and associated images she may also re- 
quest associated pathology images and reports. De- 
pending upon user preference, these "stacked" images 
and reports might be automatically retrieved along 
with the explicitly requested data. 

We are eager to ensure that the dynamic links be- 
tween Image Engine and MARS are implemented in 
a way that does not preclude Image Engine linking 
in the future to other important clinical information 
systems. One method of ensuring this independence 
may be by defining a communications layer between 
Image Engine and the other databases. 

CLINICAL APPLICATIONS 

As part of our HPCC contract, we plan to install and 
evaluate Image Engine workstations in three clinical 
environments at UPMC — clinical pathology, gastro- 
enterology, and medical oncology. 

In clinical pathology, we will focus on issues in- 
volved in digitizing, compressing, indexing, storing, 
and retrieving both gross and microscopic pathology 
images. Clinicians in the gastroenterology and on- 
cology test sites will identify pathology specimens 
(for example, biopsies obtained during gastrointes- 
tinal endoscopy or diagnostic oncology procedures) 
from patients entered into the Image Engine database. 
These pathology specimens will be digitized, in- 
dexed, and added to the database. Image Engine will 
automatically integrate these pathology images with 
other digitized images (such as endoscopy and radi- 
ology) and the text-based clinical record for that pa- 
tient. 

In gastroenterology, we will work with clinicians 
specializing in fiber-optic endoscopy of the gastro- 
intestinal and biliary tracts. These domains will in- 
volve digital still and video images, pathology im- 
ages, and radiological imaging studies (including MRI, 
CT, and ultrasound). For patients entered in the Im- 
age Engine database, it will be possible for clinicians 
to selectively view and manipulate integrated image 
and textual data. 

In medical oncology, we will explore how one man- 
ages and integrates the wide range of images (in- 
cluding radiology, MRI, CT, pathology, and clinical 
photography) that are used in the diagnosis, staging, 
and treatment of patients with solid tumors. Clini- 
cians will be able to rapidly retrieve both image and 
textual data for patients entered into the Image En- 
gine database. 

Image Engine should be useful in many other im- 
age-intensive clinical domains such as dermatology 
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[37] and ophthalmology [38]. In addition, as the num- 
ber and variety of images in the database increases, 
it has the potential to become a valuable educational 
and research resource. For example, it would be pos- 
sible for one to retrieve and view a set of pathology, 
dermatology, ophthalmology, endoscopy, and radi- 
ology images for a given disease entity. Alternatively, 
one could retrieve selected images from a population 
of patients with certain characteristics. 

It has been estimated that medical imaging data- 
bases acquire in excess of one terabyte of information 
per year in a major hospital [39], Given the large size 
of even compressed digital images, we have chosen 
to limit our initial HPCC evaluation domains to a 
relatively small population of clinicians and patients. 
Our goal in this project is not to implement a very 
large-scale, hospitalwide, image database system dur- 
ing this three-year project Also, Image Engine is not 
intended to compete with or replace PACS, which 
we see as continuing to develop as a radiological sup- 
port system. Instead, we plan to focus on the technical 
and clinical issues involved in creating a potentially 
portable system that could be scaled to handle the 
image storage, retrieval, and sharing needs of clini- 
cians, with an emphasis on integrating a wide range 
of clinically important images with the text-based 
patient record using relatively inexpensive high-per- 
formance computers and networking technology. 
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